A software-defined data center may include a control cluster that “centralizes” certain functions with a relatively small, distributed cluster of nodes to manage the data center components. One of the problems faced by the control cluster is how to manage its membership in simple, reliable, and easy-to-implement way.
A cluster should enforce a consistent membership view on all nodes, including after the cluster dynamically adds and removes nodes. Many existing clusters use coordination services, such as Zookeeper, to handle cluster reconfiguration. Unfortunately, these coordination services also need to form their own clusters and therefore face the very same problems as to how to maintain a consistent membership view. For dynamic configuration, these coordination services have strict requirements and limitations when used in practice. For example, Zookeeper requires manual recovery of the cluster if the coordination service loses its quorum. However, customers do not wish to manually manage a cluster's membership and expects this task to be automated.